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TECHNICAL REPORT 



The 

RICIS 

Concept 


The University of Houston-Clear Lake established the Research Institute for 
Computing and Information systems in 1986 to encourage NASA Johnson Space 
Center and local industry to actively support research in the computing and 
information sciences. As part of this endeavor, UH-Clear Lake proposed a 
partnership with JSC to jointly define and manage an integrated program of research 
in advanced data processing technology needed for JSC’s main missions, including 
administrative, engineering and science responsibilities. JSC agreed and entered into 
a three-year cooperative agreement with UH-Clear Lake beginning in May, 1 986, to 
jointly plan and execute such research through RICIS. Additionally, under 
Cooperative Agreement NCC 9-16, computing and educational facilities are shared 
by the two in stitutions to conduct the research. 

The mission of RICIS is to conduct, coordinate and disseminate research on 
computing and information systems among researchers, sponsors and users from 
UH-Clear Lake, NASA/JSC, and other research organizations. Within UH-Clear 
Lake, the mission is being implemented through interdisciplinary involvement of 
faculty and students from each of the four schools: Business, Education, Human 
Sciences and Humanities, and Natural and Applied Sciences. 

Other research organizations are involved via the “gateway” concept. UH-Clear 
Lake establishes relationships with other universities and research organizations, 
having common research interests, to provide additional sources of expertise to 
conduct needed research. 

A major role of RICIS is to find the best match of sponsors, researchers and 
research objectives to advance knowledge in the computing and information 
sciences. Working jointly with NASA/JSC, RICIS advises on research needs, 
recommends principals for conducting the research, provides technical and 
administrative support to coordinate the research, and integrates technical results 
into the cooperative goals of UH-Clear Lake and NASA/JSC. 
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STATISTICAL DATA FROM TASK RATING WORKSHEETS 









STATISTICAL ANALYSIS OF NASA/JSC/SSD TASK RATINGS 


DATA COLLECTION 

Fourteen questionnaires were collected from individuals employed at NASA. Thirty different 
jobs or tasks were defined in which individuals rated their perceptions of these jobs according to 
five different response variables. The ratings from these 14 questionnaires were coded into a data 
base system so that statistical analyses could then be performed. Table 1 lists the thirty jobs/tasks 
with their appropriate coding values. Table 2 tabulates data on the five response variables used 
during the interview and questioning of the NASA employees. These five response variables are 
(1) task frequency, (2) percent of time on task, (3) task importance, (4) learning difficulty, and (5) 
personnel. Codings used to transcribe these responses into the data base are also included in Table 
2 . 


Not all individuals interviewed provided answers to every job/task and every response variable. 
Therefore, the data gathered in this study contains some jobs/tasks with relatively few responses. 
These jobs/tasks cannot be used in any formal statistical procedure since their sample size is too 
small. In general, jobs which contained fewer than 10 replies ( <70 percent of the sample) were not 
included in formal statistical summaries. These are job/task numbers 5, 9, 12, 13, 18, 23, 24, 25, 26, 
27, 28, and 29. 

Pairwise Correlations 

To determine if any of the five response variables were similar or perceived as reflecting the 
same type of reply from each of the individuals interviewed, a pairwise correlation coefficient, r, was 
calculated among all possible pairs of the five response variables. These results are listed in Table 
3. It can be seen that there is no predominant linear correlation among any two of the response 
variables, indicating that the response variables are indeed measuring different factors or concepts. 
As can be seen from Table 3, the highest correlation exists between task frequency and percent of 
time on task (-0.2947). 

Descriptive Statistics 

As a synopsis of the data collected, descriptive statistics were calculated for each job/task by 
response variable. These statistics are summarized in Table 4 and include the sample size (number 
of responses), the average response rating, and the standard deviation of the response rating. A 
frequency table of the responses was also generated and appears in Table 5. 

Of the job/tasks which contained at least ten replies, the job/task performed the most fre- 
quently was 8, interact with other branch members. This response variable had an average reply 
of 1.6 which translates to once a day or more than once a day. Individuals spent at least 30 percent 
of their time interfacing with contractors (average *3.57), which they also claim is very important 
(average = 4.36). Analyzing test results is the hardest task to learn (average = 3.54) and the job which 
everyone agrees is done on an individual basis is the evaluation of contractors. Interacting with 
branch members utilizes a team effort (average* 1.08). 

Table 6 is a summary of the percentage of replies by response variable received for a particular 
job/task. Note that only jobs which contained at least ten replies are listed. 



TABLE 6. Percentage of Replies for Each Rasponsa 
Variabla by Job/Task 


Job/Task Conclusion 


1: Review Task Frequency 

57% 

Once a week to once a month 

SW Requirements 

29% 

Once a day 


14% 

More than once a day 

% Time on Task 

50% 

0 - 10% 


21% 

10 - 30% 


29% 

30 - 50% 

Task Importance 

50% 

Very important 


43% 

Important 


7% 

Somewhat important 

Learning Difficulty 

69% 

Hard to learn 


23% 

Easy to learn 


8% 

Very hard to learn 

Personnel 

71% 

Individual 


29% 

Teamwork 

2: Research Task Frequency 

50% 

Once a week to once a month 

SW Requirements 

25% 

Once a day 

17% 

More than once a day 


8% 

Twice a year 

% Time on Task 

50% 

0 - 10% 


25% 

30 - 50% 


17% 

10 - 30% 


8% 

70 - 100% 

Task Importance 

50% 

Very important 

42% 

Important 


8% 

Somewhat important 

Learning Difficulty 

91% 

Hard to learn 

9% 

Easy to learn 

Personnel 

75% 

Individual 


25% 

Teamwork 

3: Interface Task Frequency 

50% 

More than once a day 

with Contractor 

29% 

Once a day 


21% 

Once a week to once a month 

% Time on Task 

57% 

10 - 30% 


21% 

50 - 70% 


7% 

0 - 10% 


7% 

30 - 50% 

- 

7% 

70 - 100% 



Task Importance 


86% Very important 
14% Important 


Learning Difficulty 
Personnel 

4: Attend Task Frequency 

Board Meetings 

% Time on Task 

Task Importance 
Learning Difficulty 

Personnel 

6: Attend Task Frequency 

Additional 
Board Meetings 

% Time on Task 

Task Importance 
Learning Difficulty 
Personnel 


64% 

Easy to learn 


22% 

Hard to learn 

HH 

7% 

Very easy to learn 


7% 

Very hard to learn 


54% 

Individual 

1111 

46% 

Teamwork 


72% 

Once a day 

- 

21% 

Once a week to once a month 

w 

7% 

More than once a day 

— 

46% 

10 - 30% 

m 

31% 

0 - 10% 


15% 

30 - 50% 

— 

8% 

50 - 70% 

m 

64% 

Very important 


36% 

Important 

= 


• 

50% 

Hard to learn 


22% 

Easy to learn 

= 

14% 

Very easy to learn 

w 

14% 

Very hard to learn 


71% 

Teamwork 

= 

29% 

Individual 


58% 

Once a week to once a month 


17% 

Twice a year 

■ 

8% 

More than once a day 


8% 

Once a day 

== 

8% 

Once a year to less than 



once a year 


58% 

0 - 10% 


17% 

10 - 30% 

* 

8% 

30 - 50% 


8% 

50 - 70% 


8% 

70 - 100% 

- 

58% 

Very important 

— 

25% 

Important 

1 

17% 

Somewhat important 


50% 

Hard to learn 

— 

33% 

Easy to learn 


17% 

Very hard to learn 



50% Teamwork 
50% Individual 


Task Frequency 


7 : Prepare 

FSW Management 
Forms 


46% Once a day 

46% Once a week to once a month 

8% Twice a year 


% Time on Task 


54% 10 - 30% 

30% 30 - 50% 

8 % 0 - 10 % 
8% 50 - 70% 


Task Importance 


69% Very important 
15% Important 
8% Somewhat important 
8% Not important or not nec- 
essary 


Learning Difficulty 69% 

23% 

8 % 


Hard to learn 
Very hard to learn 
Easy to learn 


Personnel 


77% Individual 
23% Teamwork 


8 : Interact 
with Branch 
Members 


Task Frequency 

57% 

43% 

% Time on Task 

50% 

29% 

21% 

Task Importance 

57% 

43% 

Learning Difficulty 

64% 

36% 

8% 

Personnel 

92% 

8% 

Task Frequency 

92% 

8% 

% Time on Task 

62% 

38% 

Task Importance 

62% 

15% 

15% 

8% 


Once a day 

More than once a day 

10 - 30% 

0 - 10 % 

30 - 50% 

Very important 
Important 

Easy to learn 
Very easy to learn 
Easy to learn 

Teamwork 

Individual 

Once a week to once a month 
Twice a year 

10 - 30% 

0 - 10 % 

Very important 
Important 

Somewhat important 
Not important or not nec- 
essary 


10: Support 
SASCB 



Learning Difficulty 
Personnel 

11: Participate Task Frequency 
in Working 
Groups 

% Time on Task 
Task Importance 

Learning Difficulty 
Personnel 

14: Rev iew Task Frequency 

Test Plan/Spec 

% Time on Task 
Task Importance 

Learning Difficulty 
Personnel 

15: Review Task Frequency 

Verify Test 
Procedures' 


38% 

Easy to learn 


38% 

Hard to learn 


23% 

Very hard to learn 


69% 

Individual 

- 

31% 

Teamwork 


80% 

Once a week to once a month 


10% 

Twice a year 

■ 

10% 

Once a year to less than 



once a year 


70% 

o - 10% 


20% 

10 - 30% 


10% 

50 - 70% 

i= 



* 

33% 

Very important 


33% 

Important 


22% 

Not important or not nec- 
essary 

w 

11% 

Somewhat important 

9 

56% 

Easy to learn 

9 

22% 

Hard to learn 


11% 

Very hard to learn 


11% 

Very easy to learn 

— 

78% 

Teamwork 

= 

22% 

Individual 

■ 

70% 

Once a week to once a month 


30% 

Once a day 

m 

50% 

0 - 10% 


40% 

10 - 30% 

— 

10% 

30 - 50% 

m 

60% 

Very important 


30% 

Important 

fE 

10% 

Not important or not nec- 


;; - - 

essary 

s 

506 

Hard to learn 

5 

30% 

Easy to learn 


20% 

Very hard to learn 

n 

60% 

Teamwork 


40% 

Individual 

— 

83% 

Once a week to once a month 


17% 

Twice a year 



% Time on Task 


58% 10 - 30% 

42% 0 - 10% 


Task Importance 


58% Very important 
33% Important 
8% Somewhat important 




Learning Difficulty 

83% 

Hard to learn 




8% 

Easy to learn 




8% 

Very hard to learn 



Personnel 

67% 

Individual 




33% 

Teamwork 

— 

16: Review 

Task Frequency 

83% 

Once a week to once a month 


Verify Test 


17% 

Twice a year 


Cases 






% Time on Task 

58% 

10 - 30% 




25% 

0 - 10% 




17% 

30 - 50% 



Task Importance 

75% 

Very important 





17% 

Important 




8% 

Somewhat important 



Learning Difficulty 

58% 

Hard to learn 

“ 1 



33% 

Very hard to learn 

— 



8% 

Easy to learn 



Personnel 

50% 

Individual 





50% 

Teamwork 


17: Analyze 

Task Frequency 

90% 

Once a week to once a month 


Test Results 


10% 

Twice a year 



% Time on Task 

45% 

0 - 10% 




45% 

10 - 30% 

— 



10% 

30 - 50% 



Task Importance 

55% 

Very important 




27% 

Somewhat important 




18% 

Important 



Learning Difficulty 

55% 

Very hard to learn 

— 



45% 

Hard to learn 



Personnel 

55% 

Teamwork 




45% 

Individual 



19: Maintain 

Task Frequency 

31% 

Once a year to less than 


Current 



once a year 

Eg 

Professional 


31% 

Twice a year 


Knowledge 


23% 

Once a day 

133 

■■ — 

- "■ 

15% 

Once a week to once a month 



% Time on Task 


54% 0 - 10% 

46% 10 - 30% 


Task Importance 


62% Very important 
31% Important 
8% Somewhat important 


20 : 


21 : 


Monitor 

Processes 


Attend 
CIP and Cl 
Meetings^ 


Learning Difficulty 50% 



25% 

17% 

8% 

Personnel 

92% 

8% 

Task Frequency 

64% 

18% 

10% 

10% 

% Time on Task 

45% 

27% 

9% 

9% 

9% 

Task Importance 

55% 

27% 


9% 

9% 

Learning Difficulty 

50% 

40% 

10% 

Personnel 

50% 

50% 

Task Frequency 

79% 

21% 

% Time on Task 

72% 

21% 

7% 

Task Importance 

43% 

29% 

14% 

14% 


Hard to learn 
Very hard to learn 
Easy to learn 
Very easy to learn 

Individual 

Teamwork 

Once a week to once a month 
More than once a day 
Once a day 
Twice a year 

0 - 10 % 

10 - 30% 

30 - 50% 

50 - 70% 

70 - 100% 

Very important 
Not important or not nec- 
essary 

Somewhat important 
Important 

Easy to learn 
Hard to learn 
Very easy to learn 

Individual 

Teamwork 

Twice a year 

Once a year to less than 
a year 

o - 10% 

10 - 30% 

30 - 50% 

Very important 
Important • 

Somewhat important 
Not important or not nec- 
essary 









Learning Difficulty 

54% 

Easy to learn 




31% 

Hard to learn 

— 



15% 

Very easy to learn 



Personnel 

62% 

Teamwork 




38% 

Individual 


22: Evaluate 

Task Frequency 

64% 

Once a week to once a month 


Contractors 


36% 

Twice a year 



% Time on Task 

57% 

0 - 10% 




29% 

10 - 30% 




14% 

30 - 50% 



Task Importance 

57% 

Very important 




29% 

Important 




14% 

Somewhat important 



Learning Difficulty 

77% 

Hard to learn 




15% 

Very hard to learn 




8% 

Easy to learn 

.-W 


Personnel 

100% 

Individual 


30: Other 

Task Frequency 

45% 

Once a week to once a month 


Support Tasks 


36% 

Once a day 

~ 



9% 

More than once a day 



% Time on Task 

45% 

10 - 30% 




27% 

0 - 10% 




18% 

30 - 50% 




9% 

50 - 70% 

— 


Task Importance 

64% 

Very important 




27% 

Somewhat important 




9% 

Important 



Learning Difficulty 

80% 

Hard to learn 

— 



10% 

Very hard to learn 

w 



10% 

Easy to learn 



Personnel 

70% 

Individual 

— 



30% 

Teamwork 






s 
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■ AutoLib Product Overview. Barrios, July 1989. 

■ Configuration Control During OBS FSW Implementation. R. De la Fuente, June 1989. 

■ CR Selection Process for GN & C LVL7 Testing. T. Talbert, November 1988. 

■ Data Processing System (Hardware and Systems Software). NASA, April 1986. 
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■ Electronic Library Project. Ken Jenks, October 1987. 
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1988. 
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■ 01-8C Flight Software Program Notes and Waivers, Addendum 2. 
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date unknown. 

■ On Line Management Information Network for the Shuttle Avionics Software Control Board 
User’s Guide and Requirements. IBM, May 1988. 

■ Overview of the IBM Level 6 Verification Process. J. Haugh, July 1989. 

■ Program Notes & Waivers (OI-8C). NASA, October 1989. 

■ Shuttle Avionics Flight Software Integration Testing Report on Tools & Methods. August 1987. 
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■ Systems Management/ Payload Functions. M. Creech, May 1988. 

■ System Software Functional Overview. M. Somers. April 1988. 

a Test and Operations Procedures and Support Software Reference. NASA, June 1989. 
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■ BFS Certification, SSD. Author and date unknown. 

■ BFS Integrated Software Change Control Management System with the Software Management 
Utility. D. Eldridge, date unknown. 

■ BUS Reconfiguration Concepts/ Effects. M. Queener, July 1988. 

■ Entry Flight Control Overview. A. Currie, January 1989. 

■ Downlist Verses Downlink. D. Pipkins, November 1987. 

■ FEID Modification Overview. M. Briedon, 1989. 

■ Guidance (Ascent Phase). Pipkins & Perera, November 1989. 

■ Integrated Software Engineering Education and Training Plan. Glenn Freedman, August 1989. 

■ Level 6 Process Overview. Earl Lee, July 1989. 

■ Library Management System Presentation. Barrios Technology, October 1989. 

■ Mission Control Center Overview. S. Seyl, date unknown. 

■ Mission Operations Directorate - Operations Division Electronic Library Project. Ken Jenks, 
October 1989. 

■ 1-Load Reconfiguration Process Overview of PASS and BFS Processes. D. Eldridge, May 1988. 

■ PASS Certification. D. Pipkins, February 1988. 

■ PASS Architecture. J. Newman, January 1988. 

■ SSIP & ICC Overview. Stephenson/E. Fountain, April 1989. 

■ Systems Management/Payload Functions. M. Creech, May 1988. 

■ Tools CRs and DRs Process Overview. D. Eldridge, February 1988. 

■ Vehicle Utility Functional Capabilities. S. Dean, May 1988. 


SAMPLE HANDOUTS/DOCUMENTS 

■ DR Board Summary & Handout. 

■ FR2 Weekly Meeting Schedule, week 1-2. 

■ IBCB Flight Products Listing (Example), Sharyl Butler. 

■ Pass Status (Example), "Automated" OI Tool (Caperton). 

■ STS MER Schedule (Example) SSD, Stephenson. 



TRAINING REFERENCES 


■ Ascent Guidance and Flight Control Training Manual. NASA, August 1989 

■ Bus Reconfiguration CBT. Barrios, November 1989 

■ DPSD Courses for JSC. June 1989. 

■ Existing SMAP Courses. 

■ FR2/SSD/MSD Training. Joe L. Seale. February 1988. 

■ Guidance, Navigation and Control CBT. Barrios, December 1989. 

■ MOD/FCOD Training Materials Library Inventory. Newman, June 1989. 

■ Mod Phase I Program Guide. 

■ NASA Software Management and Assurance Training Courses. 

■ Project Management, SMAP. 

■ Proposed SMAP Courses. 

■ Quality Assurance, SMAP, E. Fountain. 

■ Software Acquisition MGMT Course Summary. 

■ Software Configuration Management. SMAP, E. Fountain. 

■ Software Verification & Validation. SMAP, E. Fountain. 

■ SSW Training Class. M. Briedon, December 1988. 

■ Summary of MOD Phase I Orientation. E. Fountain, April 1988. 
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SURVEY FORMS 



INTERVIEW SURVEY INSTRUMENT FOR 
NASA/JSC/SSD PROGRAM ANALYSTS 


Name of interviewer: — 

Date of Interview: 

1. BACKGROUND INFORMATION 

a. Name: 

b. Job Title/Grade: — 

c. Phone Number: 

d. Number of Years in present jod: 

9. Previous NASA job positions and length of employment m eacn: 


f. Previous job experience related to current NASA position: 


g. Formal educational degree(s): - 

h. Name of supervisor: 

i. Number of personnel you supervise: 

j. Personnel you interact with most frequently: 


2. JOB/TASK ASSIGNMENTS AND WORK SPECIFICATIONS 
a. Please descnbe your job in general terms. 


b. Co you have a set schedule? ■ Cay-Dy aav or weeK-oy-week ) 9 '? yes. piease describe. 


c. What Boards are you involved with and in what capacity 9 


3. TRAINING SUPPORT 

a. How were you trained for the special knowledge and skills necessary for your |0b 9 


b. What were the strengths ano weaknesses cf this training 9 


c. Do you currently receive training for new 10b assignments? if yes. please describe strengths and 
weaknesses of that training. - — 


d. How much time do you spend in training each month?. 


e. How much time could you spend in training each month?. 


f. if you had to hire a new employee to do your job. what qualities would you look for? 


mo* ■ 

g. What kinds of things could be provided to help this person understand how to do the job quickly 
" and prnfiripnily 9 — ~ 


h. Would you like to receive more training m your current job position? if yes. please elaborate. 


INTERVIEW SURVEY INSTRUMENT FOR 
NASA/JSC/SSD SUPERVISORS 


Name ol Interviewer: 

Date of Interview: — . — 

1. BACKGROUND INFORMATION 

a. Name: — 

b. Job Title/Grade: 

c. Phone Number: 

d. Number of years in present job:_ . 

e. Previous NASA job positions and length of employment in each: 


f. Previous job experience related to current NASA position: 


g. Formal educational degree(s): 

h. Name of supervisor: 

2. CHARACTERISTICS OF PERSONNEL SUPERVISED 

a. Number of personnel you supervise: 

b. Personnel you interact with most frequently: 


c. General education of personnel you supervise: 

d. General skills and abilities of personnel you supervise: 


e. General attitude of personnel you supervise towards jobs: 


Where and how do personnel currently receive training? 


k. Does NASA currently provide training? 


I. Can or do personnel receive training from other sources? 


m. What do you perceive is the major problem with the software maintenance procedures as they 
stand today? 


1 . If you perceive that there is a problem, what do you think is the cause? 


2. How should the system be working? 


3. What should be happening that is not happening? 


4. What do you think are the options for solving the problem? 


n. What is the availability of resources (hardware, software, reference materials) for your staff? 



3. JOB PERFORMANCE OF PERSONNEL 

a. is their performance adequate, lair, poor, excellent? 

b. What do they do best? 



c. If what areas (job tasks) do they need the most improvement?. 



d. What are the major constraints that seem to limit their performance? 



e. Any other problems that keep your staff from performing adequately? 



f. What positions (if any) do you have the most trouble keeping adequately staffed? 



g. What problems do you think could be eliminated by training? — 



<m <n *r in 


APPENDIX D 


FOURTH GENERATION LANGUAGE, A POWERFUL, LONG-TERM 
SOLUTION TO MAINTENANCE COST 


The attached article was written by Mr. Paul Breaux, a research analyst at SwRI, and accepted for 
publication in the Government Computer News. Before coming to SwRI, Mr. Breaux was the Chief 
of Configuration Management (CM) for the Joint Chiefs of Staff (JCS) in Washington, D C. In this 
position, Mr. Breaux established all configuration management policies and procedures for the JCS. 
He was instrumental in establishing a near paperless office automation environment in the JCS using 
available 4th Generation Language tools and networking technology. The current JCS office 
automation environment allows full coordination of message traffic, publications, and documentation 
including red-line changes without requiring hard copies. E-mail, the interactive scheduling of 
conference facilities and meetings, business graphics, spreadsheet services, individual tickler support, 
bulletin board services, distributed data basing, archival support, interactive financial transaction 
support, and complete configuration management support including interactive problem reporting, 
analysis, tracking, and CM Board administrative support. 



Fourth Generation Languages, A Powerful, Long-Term Solution to Maintenance Cost 


Currently, it is estimated that 40 to 80 percent of information systems costs are directly attributed 
to maintenance. In addition, results of numerous surveys indicate that 60 to 70 percent of DOD 
software dollars are spent annually on software maintenance. Everyone can agree that the cost of 
information system’s maintenance can adversely affect what programs are changed and when changes 
are accomplished. The following is a brief on what is a Fourth-Generation Language (4GL), benefits 
gained by using 4GLs, and some of the limitations of using 4GLs. 

4GLs were designed to meet two specific objectives. First, to assist nonprogrammers in generating 
software applications without learning more complex conventional high-level programming languages. 
Second, to assist in increasing the productivity of the normal programming process. They are meant 
to be languages that are easy to learn and simple to use by nonprogramming individuals. 4GLs are 
usually tied to specific functions such as graphics languages. Data Base Management Systems (DBMS), 
and report/application generators. They give the user a somewhat less complex method of setting 
screen displays and report formats. 

The use of 4GLs can reduce the traditionally lengthy software modification cycle. The design of 
most 4GLs allows users to find the area needing change and with minimal key-strokes make the change 
in an expeditious manner. Well-designed 4GLs will normally allow 80 percent of routine changes to 
be made in one hour, whereas, it may take months using conventional methods of programming. 

Also, since most applications developed using 4GLs are easy to understand, different people from 
those who initially programmed the application are able to make changes to existing code within a 
short period of time and with minimal frustration. 

Most 4GLs are designed to prevent ill-structured program constructs which could cause problems 
after the application is implemented. A good 4GL will walk the user through the sequence of 
operations in a clear, concise, structured manner and generate English-like source code into 
transparent object code. The user does not have to worry about learning structured programming 
constructs to get the job done. 

Some 4GLs provide automatic and/or selectable error trapping. It is nearly impossible to identify 
and provide error trapping on every possible type of error that may occur within a program using 
conventional program methodology. However, many 4GLs provide automatic error trapping or allow 
the user to identify specific error conditions and/or provide error messages to be tailored to individual 
user needs. 

Many 4GLs are self-documenting. For example, some 4GLs insert comments into existing source 
code as the application is being generated. This means that users do not have to take time away from 
normal duties to document software that is being developed or modified. In addition, the 
documentation can be used for future modifications to existing applications. 

Due to the ease of use of most 4GLs, users are able to make their own enhancements, especially 
to screen displays and report formats. This will allow the programming staff to use valuable and 
expensive application development time for more complex applications. 


In some cases, end-users are able to generate and maintain entire applications. Some 4GLs are 
so user-friendly that most end-users, with time and a desire to learn a 4GL, can generate and maintain 
simple applications. Again, this relieves programming staff for more complex programming tasks. 

Many 4GLs are either linked to or are part of a DBMS. The applications are tied to self- 
generating data dictionaries which allow them to make full use of existing data bases. Data bases are 
becoming very common in the work place. A well-designed 4GL linked to a DBMS allows users to 
design and build screen displays and generate reports that can increase the efficiency of most 
organizations. 

Although the use of 4GLs allows users to develop applications faster and provide more cost 
efficient maintenance, there are some limitations. The following limitations should be considered when 
purchasing and/or using 4GLs. 

For example, some applications are too complex for 4GLs. Even with a high-level programming 
language interface, some applications should not be attempted using 4GLs. 4GLs are best used with 
simple, routine applications required for reporting specific information (usually tied to stored data). 

Depending on the design of the 4GL, response time may be slow. Because 4GLs are designed to 
make the job of creating and maintaining applications easier, it carries some overhead. This overhead 
is additional code that is generated each time an application is developed or modified. This additional 
code is generated whether it is used by the application or not. This overhead may effect the response 
time regarding data base queries, graphic presentations, and report generation. Responsiveness should 
be considered when selecting a 4GL for use in application development. 

As related in the previous paragraphs, there is overhead witli 4GLs and this may affect the amount 
of internal and external storage space used by applications. 4GL applications tend to take up 
approximately one half to twice as much storage space as required for conventional applications. If 
there is limited storage space, care must be taken when using 4GLs. 

Some 4GLs are not general purpose enough to meet most routine application requirements. There 
exist 4GLs that are designed to meet specific functional requirements such as graphics, report 
generation, query operations, or application generation only. Based on user requirements, the 4GL 
selected should be a combination of these functions with a seamless (if possible) interface between 
complementary functions. 

Finally, the use of 4GLs can breed complacency regarding data administration. Some users believe 
that because some 4GLs provide well-structured application software, there is no need to model or 
efficiently design data structures that are required for use with the 4GL. Poorly designed data 
structures could adversely affect the performance of the 4GL. Care must be taken with data 
administration in order to get the most from a 4GL. 

An example regarding the use of 4GLs can be found at the Joint Chiefs of Staff (JCS). Currently, 
the JCS uses the PACE DBMS on Wang minicomputer systems. On the average, six applications are 
developed each month that would normally take three months to develop using conventional 
programming languages. Most of these applications are developed in less than two weeks. The 
majority of applications deal with user-friendly, form-like screens tied to stand alone data bases. 
However, with appropriate access authorization, these applications can be used by remote users 
through the Local Area Network (LAN). 



In conclusion, this brief has provided eight reasons how 4GLs can decrease software development 
and maintenance costs and some limitations to consider when selecting or using a 4GL. This brief 
should establish that even with some limitations, it is still worth any organization’s time and money to 
look into using 4GLs for many of their routine software application development 
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ANALYSTS TASKS 


Process DR/CR 

Review SW Requirements (CR/DR) 

Research SW Requirements 

Interface with Contractor Personnel (Disposition/Severity) 

Attend Board Meetings (PASS & BFS DR) 

Chair Boards 

Attend Additional Board Meetings 
Prepare FSW Mgmt forms 
Interact with other Branch Members 
Support Pre-SASCB 
Support SASCB 

Participate in working groups and mode teams 

Produce Program Notes on Waiver’s Document (each OI & Flight) 

Testing 

Develop Scenario 
Review Test Plan/Specification 
Review Verification Test Procedures 
Review Verification Test Cases 
Analyze Test Results 

Support Tasks 

Principal Function Managers (IPL, DCP, BSS) 

Certification Manager 
DR Manager 
OI Manager 
Mission Manager 
Backup/Substitute 
Flight Support 
Release Manager 
T&O Manager 

Produce Support Tools Document (once every OI) 

Book Manager 

General Tasks 

Control Documentation - Publish of Design Documents 
CPDS(IBM), PRD(RockweU) 

Maintain Current Professional Knowledge 

Monitor Processes - Development Plan Review Meetings (Schedule Status & 
Integration Planning) 

Attend CIP and Cl Meetings 
Evaluate Contractors 
Develop Training/Presentations 

Make Technical Presentations to Upper/Mid Management 
Manage Facility 
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METHODOLOGY FOR STUDY 


Analysis Techniques 

Two powerful analysis techniques were used in performance of the TNA effort for the Flight 
Software Branch: needs assessment analysis and subject matter/task analysis. 

Needs Assessment Technique 

Needs assessment is the systematic effort that SwRI followed to gather opinions and ideas from 
a variety of sources on performance problems. The needs assessment technique was the primary 
vehicle SwRI used to acquire information which was crucial for seeking a clear picture of the analysts’ 
opinions and conceptualization of the problems they encounter. 

Subject Matter/Task Analysis Technique 

During subject matter analysis, SwRI researchers sought to identify the nature and shape of bodies 
of knowledge which the analysts must have in order to perform their jobs effectively. With subject 
matter analysis, SwRI attempted to draw out the details, shape, and relationships of what is knownand 
done in order to do the job well. This information was then converted into representations of the 
optimals which are central to TNA. SwRI conducted a subject matter analysis through interaction with 
FSB analysts and managers, and through reviews of system documentation. 


Detailed Methodology for Conducting the Study 

Develop Data Collection Instruments and Strategy for Data Collection 

During this project event, SwRI researchers met initially with Mr. Robert Hinson, the Software 
Development Section head, to strategize how the data collection effort would proceed. A decision was 
made that all of the analysts and managers in the Flight Software Branch would participate in the 
study. During this initial meeting, Mr. Hinson provided SwRI researchers with a high-level description 
of the work environment and tasks which were performed by the SSD analysts. Also during this initial 
meeting, SwRI researchers observed a Discrepancy Report (DR) Board meeting to observe analysts 
as they interacted with IBM contractors. 

From the high-level description and observations, SwRI analysts developed interview survey 
instruments for the analysts and management (see Appendix C for copies of the survey instruments). 
These instruments were developed specifically for use by SwRI researchers as they conducted the inter- 
views and observed the analysts in their work environment. 

Conduct Surveys and Interviews 

During this task event, SwRI researchers conducted two interviews with each SSD analyst and one 
interview with the two branch managers and branch chief. 

Perform Task Analysis 

After the initial interviews were conducted and preliminary reviews of the literature were 
completed, SwRI researchers developed a list of tasks performed by the analysts. The task listing 
appears in Appendix E, work environment description. The task listing was reviewed by Mr. Hinson, 
and the tasks were rated during the second structured interview with each analyst. 

Conduct Statistical Analysis 

Inferential and descriptive statistics were used to analyze the data collected during the needs 
assessment and subject matter analysis. The results of the analysis are provided in Appendix A. 

Prepare Final Report 

Upon completion of the site interviews/observations and an analysis of the data collected during 
the site visits, SwRI prepared a final needs assessment report. 
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INSTRUCTIONAL SYSTEMS DEVELOPMENT 


For 20 years, the military and the industry in the United States have successfully applied 
instructional systems development (ISD) in training. The ISD methodology is a process of quality 
planning, quality control, and quality improvement to ensure effective and efficient training. This 
paper presents a model of ISD that the Instructional Systems Section within the Training Systems 
and Simulators Department at Southwest Research Institute (SwRI) has adopted. 


Overview of the Model 

Instructional Systems Development applies a systems approach to solving problems of skill 
development within an organization. It is an orderly process for planning, developing, implementing, 
and evaluating instructional programs which ensure that personnel are taught the knowledge, skills, 
and attitudes essential for successful job performance. 

The ISD model (Figure 1) divides the development process into five phases: analysis, design, 
development, implementation, and control. Typically, the first phase in ISD is to analyze the job 
along with any existing training to determine new training needs, i.e., what performance 
requirements are not adequately supported. In the design phase, the developers specify desired 
outcomes of the training including the instruction to be developed and implemented. Control is 
conducted at every stage through constant evaluation and revision of the process and product of 
training. 


Instructional Systems Development (ISD) 



CONTROL 


1 




(asi) 
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Figure 1 ISD: The Model 



It cannot be overstressed how critical the analysis phase is to the success of the development process. 
In particular, the procedures used must identify all tasks performed in a job, all actions taken by the 
worker, and the acceptable standards of output or product. People must not rush into the phases 
of design and development with preconceived training needs and solutions if they expect the training 
they develop to match the job. Failure to identify critical tasks together with their correct 
performance requirements will produce incorrect or insufficient training and can result in bodily 
injury, equipment damage, and mission degradation. A careful and thorough analysis in the 
beginning will ensure that resources are not wasted on producing unnecessary or irrelevant training. 
Similarly, continuous evaluation of products throughout the process will allow for timely feedback 
and revision, and ensure a validated and effective program as the end product. 

Each of the five phases comprises several steps. The rest of this report will describe in detail the 
steps in each phase. 
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CONTROL 


In this phase, the developers collect and review data on jobs, discrepancies between performance and 
requirements, current training, and task content. They investigate the following: 


■ What tasks do the trainees perform in their jobs? 

■ Which of those tasks need training? How much? In what format? 

■ What are the job performance requirements for the tasks selected for training? 

Three types of analysis apply (see Figure 1.1). Occupational analysis identifies all jobs performed 
by the trainees and all tasks included in those jobs. Training needs assessment identifies what tasks 
require training, and where and how much is needed. Finally, a tasks analysis identifies the require- 
ments for effective task performance (e.g., skills, knowledge, equipment, conditions, and standards). 


Occupational Analysis 

Occupational analysis is a procedure for the identification of jobs performed by all trainees and the 
tasks that comprise these jobs. To manage this process, SwRI has adopted the Comprehensive 
Occupational Data Analysis Programs (CODAP), which use the survey approach and are supported 
by computers. The CODAP is a well-established tool. Research and development of the system has 
been ongoing since the late 1950$, 

The CODAP instrument collects demographic information from job incumbents and supervisors on 
questions of interest to management as well as task inventory data. The task data provide values 
for a number of factors used in the needs assessment to select tasks for training. The demographic 
information collected will be relevant to manpower and personnel problems such as projecting 
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requirements for each task 
to be trained? 




adjustments needed to maintain a skilled work force when a significant proportion of experienced 
workers are approaching retirement. Also, when a need to redesign the career ladders surfaces, 
CODAP will allow multiple job series to be included in one survey. The system analyzes data 
obtained simultaneously from different work groups to reveal similarities and differences among the 
groups. 

Among CODAP output, automated training indicators (ATI) are of particular interest to 
instructional designers. ATI is an index calculated for a task based on percentage of members 
performing it, its training emphasis, and task difficulty ratings. A training decision logic table has 
been designed to be used with ATI. For example, a task which is performed by a large percentage 
of employees and has high rating? of training emphasis and task difficulty gets a high ATI value. 
It indicates that formal knowledge and practical training should be provided for this task. 

Although the CODAP program designs are as comprehensive as is possible at this time, they will 
not produce all the data required to complete the occupational analysis. Other methods such as 
work observation and focus group discussion may have to be used to supplement CODAP. Analysts 
often watch (and sketch or photograph) work in progress to verify actual task step details. Focus 
group discussion uses groups of eight to twelve people with a moderator using an interview guide 
to obtain expert opinions on a specific topic, such as subject matter experts’ opinions on what 
training will be needed to accommodate a forthcoming technology change. 


Training Needs Assessment 

Using the complete task list generated in the occupational analysis, ISD analysts perform a needs 
assessment to ensure that some form of instruction will be provided for all the important tasks; and 
instructional resources will not be wasted. Some tasks may not require any training if all or most 
entry-level job holders can easily perform the tasks, and not all tasks require the same amount of 
training detail. 

One product of the needs assessment will be identification of the appropriate instructional format. 
Options include: 

■ classroom training 

■ structured on-the-job training (OJT) 

■ self-study 

■ reference materials (e.g., job technical manuals, performance aids, and decision 

tables) . , - — - 

The selection depends on differentiating among tasks that need to be taught (instruction required), 
those that are too easy to need instruction (information required), those too rarely performed to 
justify training (reference material required), and those already well known by the target population 
(prerequisites). It is also important to identify when to offer what training (e.g., initial vs. refresher 
training). 

The ISD model requires the analyst to identify the criteria used for establishing the need for 
training. Criteria may vary; but at a minimum, they will include the following: 
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■ Percentage of Members Performing the Task 

If a task is actually performed by most of the job holders, training is most likely to 
be justified. 

■ Criticality of the Task 

The criticality of a task relates to the seriousness of the results of inadequately 
performing a task. Failure to perform a critical task correctly could result in injury 
or loss of life, damage to equipment, and significant mission degradation. A highly 
critical task, even performed by few, is the top training priority. 

■ Task Difficulty 

Tasks relatively easy to learn on the job will need little or no formal training. For 
highly difficult tasks, a great deal of formal training may be required. 

■ Frequency of Task Performance 

A task that occurs rarely will not have a high priority for training unless it has been 
identified as having critical consequences. 

■ Time Interval before Initial Job Performance 

If analysis shows a large time interval between formal training of a certain task and 
the first time that task is performed on the job, options should be considered, such 
as postponing formal training to a later stage of the training curriculum or offering 
periodic refresher training sessions to reinforce retention and skill proficiency. 

The output of the Training Needs Assessment is a list of all job tasks selected for training together 
with information such as when and in what format training will be conducted. 


Task Analysis 

Task analysis determines the content of the training. Information includes: 

■ activities required to accomplish a task and their sequences 

■ skills, knowledge, attitudes, and physical requirements associated with the activities 

■ the equipment and materials involved with each task 

■ conditions/environment under which these tasks are performed 

■ standards for successful performance 

■ safety requirements 

Sources of data include: 

■ observing tasks being performed 

■ interviewing novice and experienced performers and supervisors 
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■ reviewing technical documentation for the tasks T . 

■ reviewing effectiveness data on current and previous training 

Results of the detailed content analysis for each task to be trained provide input into developing 
objectives and tests, defining training content, determining structure and sequence, and identifying 
appropriate training methods and media. 


Output of The Analysis Phase 

The end-product of the analysis phase is a set of training recommendations including the following 
information: 

■ tasks identified for training 

■ training formats, such as formal training, OJT, job aids/reference materials, periodic 
updates, and self-study, for each task 

■ detailed specifications of job performance requirements, including skills/knowledge, 
physical requirements, equipment, conditions, and standards, for each tasks. 


Quality Control 

To make sure the analyses use accurate and unbiased data, obtain information from a wide variety 
of sources, always checking one source of information against another. 
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Design 



CONTROL 


This phase addresses the elements that define the training approach. It begins with assessing the 
target audience. Then, learning objectives are defined and sequenced; and methods, media, and 
evaluation approaches are specified (see Figure 1.2). 


Identifying the Target Population 

Knowing the entry level skills of the target population is critical in developing effective training. The 
starting point for training depends on what the trainees already know. Target population 
characteristics will influence both the design and the delivery of training. 

Job Skills 

Prerequisite skills that trainees already possess need to be specified to attain a homogeneous target 
population and eliminate the need for offering very basic courses to level out major differences. 
Nevertheless, different trainees possess different skills and different degrees of proficiency. 
Therefore, it is useful to identify each trainee’s profile of job-related skills to minimize mismatches 
that would teach a trainee something he or she already knows or attempt to teach material he or 
she is not prepared for. 

To assess skill profiles, a global diagnostic test or a series of course-specific bypass tests can be used. 
A diagnostic test evaluates each trainee on the proficiency of all skills covered in the job. A bypass 
test, on the other hand, evaluates only skills covered in one course. Students only need to take the 
global diagnostic test once, and results give a relatively holistic picture of a trainee’s skill profile. 
Bypass tests, on the other hand, kill two birds with one stone — once the bypass tests are 
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developed, they will serve as end-of-course performance measures, which must be prepared anyway. 
Whichever is selected, the test(s) must be validated before use. 

In addition to the test(s), a supervisor’s assessment and recommendations about a trainee should 
also be considered. For a supervisor to make accurate recommendations, a detailed description for 
each course in the training curricula is essential, which includes detailed information of instructional 
objectives, content outline, and entry-level skiDs/knowledge (see Output of the Development Phase 
for details). Without this information, supervisors are less likely to make appropriate recommenda- 
tions. 

Language Skills 

The reading level of the target population must be established for written training development. 
Instructional materials must be written at trainees’ reading level. Otherwise, language skill 
requirements should be imposed in the personnel selection process to screen out unqualified 
applicants. This approach is not recommended where language skills cannot be shown to be a job 
requirement (rather than a training requirement). 


Specifying Instmctional Objectives 

An instructional objective is a precise statement of the knowledge and skills that the students must 
master. It describes what students will be able to do after instruction. A complete statement of an 
objective specifies the desired performance, the minimum standard of performance required, and 
the conditions under which the behavior is exhibited. The information is derived from data in task 
analysis. 

When properly stated, objectives will be able to serve as the basis for selecting appropriate instruc- 
tional methods and media, detailing instructional content, and developing a valid evaluation plan and 
instruments. When clearly communicated to the learners, objectives prepare them for the training 
and focus their attention on the important content. 

Supporting Objectives 

Each instructional objective is reached through a series of supporting objectives. Supporting 
objectives are intermediate steps toward a terminal objective. Some of them are included in the task 
being taught, and others are presumed to be prerequisites. Since prerequisites are not covered in 
instruction, they can be used to screen the target population of the course. Alternatively, the 
learning objectives may be revised, or optional basic courses (formal or self-study) may be offered 
to new employees. 

Classification of Objectives 

Human learning can be classified into the following five categories: 

■ verbal knowledge stating information 
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■ intellectual skill using concepts and rules to solve problems; responding to classes of 
stimuli as distinct from recalling specific examples 

■ cognitive strategy originating novel solutions to problems; utilizing various means for 
controlling one’s thinking/leaming processes 

■ motor skill executing bodily movements smoothly and in proper sequence 

■ attitude choosing to behave in a particular way 

Each category of learning requires different types of instruction. For the purpose of prescribing 
appropriate instructional treatment, it is important to classify objectives into these categories. 


Determining Presentation Sequences 
Presentation of content should follow these guidelines: 

■ proceed from the known to the unknown. 

■ proceed from the simple to the complex. 

■ proceed from the concrete to the abstract. 

■ proceed from observation to reasoning. 

■ proceed from a whole view to a more detailed view to a whole view. 

Each instructional unit should also contain the following nine events of instruction to ensure that 
desired learning outcomes occur. 

1) Ob tain lear ner’s atte ntion 

2) Inform learner of the objectives 

3) Stimulate recall of prerequisites 

4) Present the information 

5) Provide learning guidance 

6) Elicit performance 

7) Provide practice and feedback 

8) Assess performance 

9) Enhance retention and transfer 


Determining Delivery Strategies 

After instructional objectives have been written, the next step is to select appropriate delivery 
strategies — that is, instructional methods and media — to achieve the desired learning outcomes. 
Methods are procedures or processes. Media are means by which those procedures or processes are 
expressed. Examples of methods include the following: 

■ presentation 

■ demonstration 

■ drill-practice 
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■ tutorial 

■ gaming 

■ simulation 

■ discovery 

■ problem-solving 

Examples of media are: 

■ print 

■ audio tape/slides 

■ overhead transparencies 

■ filmstrips 

■ films 

■ mock-ups 

■ videotapes 

■ computer/interactive videodiscs 

■ part-task trainers 

■ simulators 

■ telephone 

■ TV 

■ satellite 

Media, depending on their design and capabilities, can be stand-alone or supplementary to classroom 
instruction. For example, some tasks may be taught in their entirety using a computer-based training 
(CBT) program. A CBT program can also support stand-up instruction. Different methods and 
media can be used to accomplish the same instructional objectives. Selection of the most 
appropriate method or medium, however, depends on many factors. 

The decisions involving methods and media selection are interrelated and should be made 
concurrently. However, for the convenience of explanation, they are discussed separately below. 

Instructional Methods Selection 

The selection of appropriate instructional methods depends on three factors: types of objectives, 
stages of learning, and instructional settings. 

Types of Objectives Earlier it was mentioned that objectives can be divided into five 
categories. These categories include: verbal information, intellectual skill, cognitive strategy, 
motor skill, and attitude. Each of these categories requires different considerations in the 
methods selection decision. If an objective involves attitudes, the use of human modeling is 
usually appropriate and effective. For verbal information, on the other hand, straight 
presentation and practice of recall may be adequate. 

Stages of Learning Any method of instruction can be used at any stage of learning, but some 
are more appropriate for one stage than another. For example, during the early stage of 
learning a skill that involves great danger, demonstration and simulation may be more 
appropriate than the drill and practice method. 
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Instructional Settings Training can take place in a large-group, small-group, individualized, 
or teletraining setting (where the instructor is physically separated from the learners). Large- 
group training may be less expensive; however, it does not accommodate individual needs. 
On the other hand, small-group and individualized settings cater to individual needs, but they 
are also more expensive. Therefore, when cost is the primary concern or when the instruc- 
tional objective is solely the delivery of information, large-group may be considered. 
Otherwise, consider small-group or individualized. 

Further, when the target population is geographically dispersed, it may be more cost effective 
to deliver training via teletraining than to bring aU the trainees to a centralized location. 
Teletraining is also effective for providing updates and refresher training. 

In selecting methods, the instructional setting needs to be considered. For example, drill/ 
practice and tutorial lend themselves to a small group or individualized setting while 
presentation or demonstration is more appropriate for a large-group setting. A combination 
of demonstration via videotape and locally supervised drill/practice may be appropriate for 
skill training. 

Instructional Media Selection 

The selection of appropriate instructional media depends on such factors as size of the target 
population, level of resemblance to the job, stability of task content, instructional methods, learner 
characteristics, media capabilities, resources/logistical requirements, and costs. 

Size of the Target Population The n umb er o f t rain ees can determine the cost-effectiveness 
of using a particular medium. For example, if an instructional unit will be offered 
repetitively to a large number of trainees, computer-based training such as interactive 
videodisc may be more cost effective than a live instructor since the cost will be scaled down 
with mass production. 

Level of Resemblance to the Job Generally, it is best to have the instructional situation as 
similar to the job situation as possible. However, high cost, safety, and limited resources 
often prevent the use of actual job environment. In general, tasks that require concrete 
actions (e.g., remove and install an engine) benefit from a realistic learning environment on 
simulation. Tasks with abstract information (e.g., compute budget estimates), on the other 
hand, require a less realistic learning situation. 

Stability of Task Content Updating training materials can be as costly as developing new 
ones. For example, it may not be cost effective to develop an expensive interactive videodisc 
program to teach tasks requiring constant updates. 

Instructional Methods Limited by presentational capabilities, certain media can only convey 
certain methods. For example, intelligent tutoring systems (e.g., intelligent computer-assisted 
instruction, interactive videodisc, expert system, etc.) are great tools for drill/practice, 
simulation, and problem-solving methods. Print material, on the other hand, is an efficient 
and cost-effective way to deliver straightforward information. It would be a waste of 
resources to use an intelligent tutoring system for the sole purpose of presenting information. 
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By the same token, print material, though less expensive to produce, would not be 
appropriate if practice of physical skills is involved. 

Learner Characteristics To select media, the following questions regarding the target 
population need to be answered: 

■ What is the range of the target population’s reading level? 

■ Is the target population computer-literate (if the computer will be used)? 

■ How comfortable is the target population with a particular medium? 

In other words, literacy is the key word. If a sophisticated medium is selected, practice must 
be provided for learners to become comfortable with that medium. 

Media Capabilities Media range from still visual (flip charts, overhead transparencies, 
printed text, etc.), still visual with audio (synchronized audio tape/slides), motion (videotape, 
film), to highly complicated computer-based training, IVD, and simulators. Each medium 
offers different capabilities of presentation. If visual recognition is involved, or if the use of 
visuals will enhance recall of the content, media with visual capability are helpful. A 
combination of media is often desirable — e.g., a videotape and a workbook, or overhead 
transparencies used in conjunction with a classroom instructor. 

Another way to categorize media is by the level of interactivity a medium allows between the 
source of instruction and the learner. With media like textbooks, film, videotape, and TV 
broadcasts, the information flow is one-way, from the instructional source to the learner. On 
the other hand, intelligent tutoring systems such as CBI and interactive videodisc are highly 
interactive in the sense that they are able to provide immediate feedback to the learner and, 
in some cases, prescribe individualized instructional treatment. Interactive media tend to be 
expensive to produce, but as mentioned above, they become cost effective with use by large 
populations over time. Trade-offs need to be made considering capabilities of the media, 
the criticality of the task, and budget. 

Resources/Logistical Requirements The availability of resources required to develop the 
program is an important selection factor, as are costs required to deliver, maintain, and 
update the training materials. 

Resources for consideration during media selection process include: 


Equipment: 

instructional 


support 

Facilities: 

classrooms 


labs 

Manpower: 

instructors 

subject-matter experts 
instructional designers 
material production crew 
administrative support staff 
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■ Time for development 

Costs Two kinds of costs are involved in training programs, development costs, and running 
costs incurred during use. The cost per student of a presentation diminishes as the number 
of students using it increases. Some media with high development costs but lower running 
costs may in the long run be more cost effective. 


Designing Evaluation Strategies 

Evaluation is crucial in the success of the ISD process. Without evaluation, the effectiveness of a 
training program cannot be demonstrated and training costs cannot be justified. Although the actual 
effectiveness of an instructional program cannot be determined until it has been implemented, it is 
important that evaluation strategies are planned out at the design stage. 


Assessing Student Performance 

The ultimate goal of evaluating an instructional program is to find out whether instruction results 
in desired knowledge/skill gain and, more importantly, improved on-the-job performance. Written tests 
are usually the instruments used to evaluate knowledge gain. Performance measures, on the other 
hand, commonly require the student to execute a task to demonstrate his or her mastery of the 
performance objectives. Whichever is used, instructional objectives are the basis on which to design 
th€|e l instruments.^:^ 

Assessing on-the-job performance (i.e., how much the skills are transferred to the job) is more 
complicated. It usually involves observing a trainee at work in a real-job situation and interviewing 
the trainee’s supervisor. 

Assessing Instructors, Media, and Student Materials , 

In addition to student performance, other elements to look at to assess the quality of an instructional 
program are: 

■ instructor subject matter expertise 

presen tatio n skills 
media utifizatioh skills 
instructional management skills 

■ medium relevancy of objectives 

accuracy of information 
technical quality 
pacing 

■ student materials textbooks 

reference materials (job aids, decision tables) 
handouts 
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other supplementary materials (organization, language level, etc.) 

Sources of data are students, subject-matter experts, instructional designers, and media experts. 
Surveys, observations, and interviews are common methods of data collection. This assessment can 
be useful in identifying weak points in an instructional program in order to pinpoint needed 
revisions. 


Analyzing Training Costs 
The costs associated with training include: 

■ 

Developmental Costs 

production 

equipment 

materials 

■ 

Implementation Costs 

equipment 

materials 

facilities 

instructional staff 
student training time 
travel 

operation/ maintenance 

■ 

Evaluation Costs 

evaluators 

materials 

equipment 

« 

Overhead 

administrative support 


Some guidelines for estimating training costs are suggested: 

1. Look at training costs in terms of the salary and loss of productivity of the learner 
while in training. The monetary value of the time spent by learners usually makes 
up the largest part of training costs — typically 80%. Learner training time is the 
single most significant training expense — not program development costs. These 
costs, however, may be offset by improved job performance. 

2. Use "full costs per hour" to estimate the labor costs of a training project. Full labor 
costs take into account salary or hourly wages plus all fringe benefits. 

3. To complete an analysis of labor costs, estimate the time spent on developing a state- 
of-the-art training program, or compare supplier price estimates with internal costs. 
The following are useful, though far from hard and fast, development time ratios: 

■ 15-40 hours of development time produce one hour of instructor-led delivery. 
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■ Between 40 and 1 10 hours of development are required to produce one hour 
of completed video. (Typically, industrial costs for outside-supplied videotape 
run from $1,000 to $2,000 per finished minute.) 

■ Approximately 100 to 400 hours of development result in one hour of 
computer-based training program. 

■ Between 200 and 1,000 hours or higher of development produce one hour of 
interactive videodisc training program. 

More complicated learning processes require additional development time. Instruction that involves 
synthesis, application, or problem-solving is more costly than instruction that calls for recognition 
or recall. 

To prepare a training budget or to estimate program costs, a spreadsheet application is 
recommended. Using a spreadsheet allows changing of variables and values for estimating purposes. 
Notice that the spreadsheet totals the costs for each step of the program: analysis, design, 
development, delivery, and evaluation. 

Output of the Design Phase 
The output of the design phase will be documents that specify 

■ the instructional objectives 

■ the course structure (modules, lessons, courses) and the content sequence 

■ the delivery strategies (methods and media) 

■ the evaluation plan 


Quality Control 

To make sure that sound design decisions are made, the entire process needs to be monitored with 
care. Criteria will be developed to assess the quality and completeness of each design document. 
A Quality Assurance Group, composed of subject matter experts, instructional designers, and 
managers, will be formed. Each major design decision will be defended or justified with rationales 
to the members of this group. • 
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Development 



CONTROL 


The development phase is where the actual production of the training course takes place (see Figure 
1.3). The process usually involves both selecting/modifying existing materials and developing new 
materials. Obtaining tangible vendor training at the time of major equipment purchase can meet 
part of the requirement. 


Selecting Existing Materials 

When selecting off-the-shelf instructional material, check off the following criteria: 

■ Does the material match the instructional objectives in the design phase? 

■ Does the target population have the prerequisites (reading ability and vocabulary 
level are often important)? 

■ Is the information accurate and up to date? 

■ Is the presentation likely to arouse and maintain interest? 

■ Does it promote active involvement of learners? 

■ Is the technical quality acceptable? 

■ Has the material demonstrated its effectiveness, such as in field tests? 

■ Is it free from bias? 

It should be noted that different criteria are suitable for different situations. It is up to the 
instructional designer to decide which criteria are the most important. 
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media utilization skills, 
instructional management skills 




Developing New Materials 


Experienced instructional developers work within the requirements of the chosen delivery media to 
tailor instruction to the terminal and supporting objectives specified in the course design. Materials 
produced usually include such items as instructor and student guides, course outlines, instructional 
content, tests, and remedial sequences. Formats will depend not only on media but on such factors 
as whether the course is classroom or self-instructional, drill and practice or simulation, process or 
product based. 


Developing Evaluation Instruments 

Following the evaluation specifications from the design stage, test items for each courses, checklists 
for skill performance, survey questionnaires, and observation recording sheets are developed. The 
tests need to be validated to ensure they measure the appropriate performance. 


Validating Instructional Materials 

The developers conduct formative evaluations during development to make sure that the 
instructional products meet all the instructional requirements specified in the design phase. Data 
collected are used to revise and improve the product under development. The advantage of this 
process is to detect errors in design and development while it is still possible to correct them. In this 
process, the instructional product is tried out by students sampled from the target population. 
Instructional designers and media experts will also examine the product. The development-tryout- 
revision cycle goes on until the students can meet the performance standards. 


Procurement of Tangible Training 

When new equipment or hardware is purchased, vendors normally provide some training, which is 
usually not documented in a tangible form such as training manuals or computer-based instruction. 
Once vendors fulfil their contracts and withdraw, there is no way to retain the training for future use. 
However, providing ISD-based training materials can be included in the procurement contract. This 
would enable the buyer to be self-sufficient in future training. Requirements for tangible training 
must be specified in the Request for Proposals for the equipment (see Output of the Development 
Phase for details). 


Obtaining Qualified Instructors 

A qualified instructor should have the following competencies: 

■ subject matter expertise 

■ delivery skills 

■ media utilization skills 

■ instructional management skills 
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Qualified instructors can be obtained through careful screening and/or training. 

Instructor preparation is especially important when unconventional instructional methods and 
technologically advanced media are used. 


Output of the Development Phase 

The end-product of the development stage may include most or all of the following components: 

■ Detailed Course Description 

instructional objectives 

target audience (skill profile, language skills, etc.) 

format of training (OJT vs. formal training, resident vs. nonresident, group- 
based vs. individual-based, teletraining vs. traditional, etc.) 
prerequisite skills/knowledge 
content outline 

■ Student Materials 

textbooks 
workbooks 
handouts 
self-study guide 

reference materials (e.g., manuals, job aids, decision tables) 

■ Instructor’s Guide 

instructor’s requirements 
lecture notes 
media utilization tips 

equipment, facilities, and scheduling in formation 

■ Instructional Material 

mediated programs (such as videotapes, computer-based training, etc.) 
instructional activities (question and answer, group discussion, etc.) and their 
duration and sequence 
drill/practice, assignments and quizzes 

■ Evaluation Plan and Instruments 

written measures (tests) and performance measures (checklists) for evaluating 

student end-of-course perfor mance --- „ 

instruments for overall assessment of the package (the instructor, media, 
student materials, etc.) by both students and instructional designers 
follow-up evaluation plan and instruments for assessing on-the-job 
performance 
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Quality Control 


A complete List of criteria should be generated against which each instructional product, selected or 
newly developed, will be evaluated. Again, a Quality Assurance Group may be one good means to 
approve instructional materials. 


Implementation 



CONTROL 


Issues to be addressed at the implementation stage include: 

■ delivering instruction as designed 

■ maintaining the quality of instruction 

■ evaluating and improving the instructional staff 

■ scheduling training to meet the demand 

■ maintaining sufficient resources (manpower, facilities, and equipment) 

■ monitoring the effectiveness of training to identify necessary revisions and updates 

As for any organizational change, the most critical factor for successful implementation of an 
instructional system is top management commitment. Managers must become believers in the ISD 
concept. This means insistence upon a good, detailed front-end analysis and a strong conviction of 
the importance of the quality control process. 


Output of the Implementation Phase 
The output of this stage includes: 

■ Coordination/scheduling of resources to meet training demands 

■ Qualified/trained instructional staff 

■ Delivered training 

■ Evaluation data (both short-term and long-term) collected 

■ Revisions made to maintain the effectiveness of training 
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DEVELOPMENT 
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Figure 1 .4 ISD: Implementation 




Quality Control 


The Quality Assurance Group can be used in monitoring and maintaining the quality of the program 
during implementation. This group makes sure that training is delivered the way it is supposed to 
be, and the evaluation-revision loop is established. 


Control 



CONTROL 


The control component of ISD is a quality assurance mechanism that involves a third-party 
validation of the analysis, design, development, and implementation of training with the purpose to 
improve instruction. Control requires the process and product of training to be constantly evaluated. 
Information collected from the evaluation is then fed back to the system to revise instruction. The 
evaluation-revision loop is essential to the quality of any training program. For example, if instructor 
performance data are not looped back to instructors, they cannot know how well (or how poorly) 
they have performed; and therefore, they will not know how to improve their teaching. If student 
performance data (both end-of-course and long-term on-the-job performance) are not analyzed, the 
design team will not know which part of instruction needs revision. 


The Quality Assurance Group 

To ensure the establishment of the evaluation-revision loop, a Quality Assurance Group must be 
formed. This group is composed of instructional designers, subject matter experts, managers, 
instructors, media specialists, apprentices, and experienced job holders. Periodically, group members 
will review the evaluation data to see if the effectiveness of training is deteriorating. If so, investiga- 
tions will be launched to identify the problems — the content may be outdated, the target popula- 
tion may have changed, instructors may not be qualified, wrong skills may have been addressed, or 
some environmental factors may exist that have interfered with learning. After problems have been 
identified, the design team is required to revise or redesign instruction. 

To stay objective, the Quality Assurance Group should be independent of the course/curriculum 
development team. That is, members of this group are not involved in the development of training, 
so they can look at instruction with fresh and unbiased eyes. 
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The Quality Assurance Group monitors not only the product but also the process. During the 
analysis phase, they make sure that data collected are complete and unbiased, and training 
recommendations are made on justifiable grounds. During the design phase, the Group reviews each 
design document on its soundness, completeness, practicality, and feasibility. It may require the 
design team to provide rationales behind each desijpi decision (e.g., why computer-based training 
is selected over stand-up classroom lecture). During the development phase. Group members make 
sure that instruction (selected, modified, newly developed, or procured from vendors) meets the 
design specifications. If it is found that certain design specifications are not feasible, they see to it 
that best alternatives are considered. Furthermore, whether instruction is developed in-house or by 
an outside contractor, the Group will ask to see hard evidence of the effectiveness of the product 
(e.g., from a field test). Finally, at the implementation stage, the Quality Assurance Group 
constantly oversees training to make sure that it is properly delivered, instructors are well prepared, 
equipment and facilities maintained, and timely updates made. 

With the evaluation-revision loop (or control), the development and delivery of training are 
constantly monitored and necessary measures are taken to fix the problems. Facilitated by the 
Quality Assurance Group, this mechanism maintains the quality of instruction and keeps training 
up to date. 



Summary 


To summarize, this report has described a systems approach to the development of training. The 
five basic components, together with their subordinate steps, represent the procedure one employs 
to ensure effective training. 

At the analysis phase, occupational analysis, training needs assessment, and task analysis are 
conducted to identify when, where, what, and how much training is needed. These analyses yield 
information concerning training curricula/paths, prescription of training formats for different tasks, 
skills/knowledge to be trained, and job performance requirements. 

At the design phase, instructional objectives are defined; and media selection is made. Evaluation 
plans for assessing the effectiveness of training are specified including end-of-course assessment and 
long-term on-the-job performance appraisal. 

At the development stage, instructional materials are produced and validated before they are used. 
Validation is a process that reiterates the development-tryout-revision cycle until the product proves 
effective. 

The implementation stage addresses the issues of delivering training and maintaining the quality of 
training. Upper-level management support is a must for the success of any training programs. 

The evaluation/revision loop (control) is a quality assurance mechanism that involves a third-party 
validation of the analysis, design, development, and implementation of training. A Quality Assurance 
Group may be used, which is composed of managers, instructors, instructional designers, experienced 
job-holders, and apprentices. Periodically, members of this group will collect and review evaluation 
data to identify deficiencies in the training system. Revisions will then be made to improve the 
programs. 

In conclusion, application of the systems approach to training development will result in improved 
job performance, elimination of unnecessary training, greater use of various training formats, and 
improved training efficiency. 
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RELEVANT RESEARCH 


APPENDIX H 

REVIEW OF RELEVANT COGNITIVE LEARNING RESEARCH 


SwRI’s solutions to identified needs are supported by cognitive-based instructional theories 
which have been hypothesized by different researchers over the last thirty years. The researchers 
have conducted experiments which empirically validated the theories. From this research, a wide 
variety of discoveries have been made about cognitive functioning. The following section 
summarizes conclusions from theoretical studies associated with adult cognition which specifically 
relate to the needs assessment and solutions recommended in this study. 


Interactive Training 

Although opinions vary among learning theorists as to what internal processes occur as an 
individual learns, there is considerable agreement that certain conditions improve learner 
performance. These conditions include active participation, practice, feedback, repetition, and 
motivation. Studies have also shown that information taken in through multiple senses is more 
readily learned than through a single sense. 

Interactive training (such as computer-based interactive videodisc) offers the learner an 
opportunity to respond to stimuli and to receive almost immediate feedback, reinforcement practice, 
and remediation as required. According to Fleming and Levi (1978), the core feature of the 
technology - interaction • is based on a sound educational principle: 

In general, where the learner reacts to or interacts with the criteria! stimulus, learning 
is facilitated and that facilitation increases with the degree of learner activity or 

involvement. 

There are other message design principles available with interactive videodisc training simply 
by the nature of the medium: 

■ Words and pictures promote better retention and learning than words alone 

■ Concepts about concrete objects are more readily learned than concepts about abstractions. 
Observing the attributes of a concept from an image and then distinguishing them visually 
from nonattributes (features not related to the concept) helps a person learn and remember 
the concept 

■ Observing a person performing some activity and then modeling the observed behavior is 
a very powerful instructional approach, especially for teaching problem solving 

More learning occurs as the user’s involvement with the prepared materials deepens, from 
simple attending (reading, looking, listening) to covert responding (repeating words subvocally) to 
overt responding (concrete actions). A well-designed, high-quality interactive training program 
encourages learners to produce more intense, overt responses. 



Spatial Representation of Information 

A number of training researchers have recognized an overemphasis on verbal skills and lack of 
emphasis on visual skills in modem education and training. These researchers have suggested that 
imagery and spatial ability skills be included in instructional design endeavors (Hotin. 1982; Galin 
and Omstein, 1979; Schroeder, 1980). Several recent studies indicate that providing subjects with 
a visual representation enhances learning and performance on some tasks. In a research program 
investigating individual differences in spatial/verbal cognitive ability, Schroeder ( 1988) showed that 
there was a general performance superiority for subjects receiving spatially-augmented information. 

Individuals in training programs are often not provided with conceptual overviews of the entire 
discipline or system they are studying. Instead, training segments are often fragmented and targeted 
for task-specific training objectives. Some students form mental models of the system from the 
fragments they have learned; however, when such abstraction is left up to the individual there are 
no guarantees that their model will be technically accurate. 

Spatial representations lend themselves to conveying general organizational information. While 
verbal descriptions can be created to describe system relationships, spatial representations can often 
be more precise and meaningful. 



Hierarchical Structuring of Instructional Content 


Cognitive psychology has recognized that the brain of the learner is the active agent in 
information acquisition rather than the passive receiver (Shriver, 1984). Instructional theorists such 
as Bruner (1956), Ausubel (1968), Anderson (1976), Norman (1975), Bobrow (1968), Reigeluth 
(1983), and Merrill (1983) discuss the effects of cognitive structures believed to reside within the 
learner on acquisition and retention of knowledge. They assume a hierarchical organization of 
knowledge in memory from abstract and general to concrete and specific. They also assume that 
semantic similarity is the basis for the linkages with which the information is related within a 
knowledge structure. Networks or schemata are the terms used by researchers to refer to the units 
of knowledge created by means of these linkages. 

The two major theorists in hierarchical structuring of instructional development are Ausubel 
(1968) and Gagne and Briggs (1974). These two theories are addressed below. 


Ausubel’s Meaningful Reception Learning and Retention 

Ausubel is primarily concerned with meaningful reception learning and the acquisition and 
retention of bodies of knowledge. Ausubel makes a distinction between reception learning, rote 
learning, and discovery learning. Rote learning involves memorization of arbitrarily associated 
bodies of information for which the learner lacks relevant prior knowledge. Discovery learning, 
involves the presentation of specific instances from which the learner must discover what is to be 
learned. In reception learning, the "what is to be learned" is presented to the learner in its final 
form. 

Ausubel advocates the use of advance organizers in instructional development. The advance 
organizer provides both an overview of the more detailed information to be presented in the 
instruction, and a scheme or organizing elements that will incorporate the new material into a 
person’s memory (cognitive structure). The advance organizer’s principal function is to "bridge the 
gap” between what the learner already knows and what he needs to know to learn the task at hand. 

Ausubel also recommends the use of comparative organizers to integrate new ideas with 
basically similar concepts already in a person’s cognitive structure and to increase a person’s ability 
to discriminate between new ideas and existing ideas that are essentially different but appear 
confusingly similar. 

With Ausubel’s theory, instructional material is organized by first presenting the more general, 
more inclusive ideas followed by more specific and less inclusive ideas. Going from the "top of the 
pyramid to the base." This process is called progressive differentiation. 


Gagne’s Conditions of Learning 

Gagne emphasizes the internal and external conditions of human learning. These conditions 
are derived from research on learning and learning efficiency. The conditions for learning are 
explicated by an information processing model of learning and memory (a model which is employed 
by a number of modern learning theorists). This model incorporates a sequence of internal 
processes involved in each act of learning and five different kinds of memory organizations which 



are internally stored as a result of learning and which exhibit themselves as types of human 
capabilities (or "learning outcomes). Both of these sets of internal conditions carry implications for 
the external conditions of learning. 

The input variables (instructional events) to Gagne’s theory are: 

1. Gain the attention of the learner 

2. Inform the learner of the objective 

3. Stimulate recall of prerequisite learning 

4. Present the stimulus material 

5. Provide "learning guidance" 

6. Elicit performance on the part of the learner 

7. Provide feedback about performance correctness 

8. Assess the performance 

9. Enhance retention and transfer 
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